Ensemble Methods – Classifier Combination in Machine Learning

The last ten years have seen a research explosion in machine learning. The rapid growing is largely driven by the following two forces. First, separate research communities in symbolic machine learning, computational learning theory, neural network, statistics and pattern recognition have discovered one another and begun to work together. Second, machine learning technologies are being applied to new kinds of problems as well as in more traditional problems such as speech recognition, face detection and recognition, handwriting recognition, medical data analysis, biometrics, data mining and game playing etc. [Jain00]. Dietterich summarized four of these directions in his review paper [Dietterich97], one of them is ensembles of classifiers, the other three are scaling-up supervised learning, reinforcement learning and learning stochastic models. Ensemble methods are learning algorithms that construct a set of classifiers and then classify new data points by taking a (weighted) vote of their predictions [Dietterich00]. It has been applied to a wide range of real problems, such as object detection and recognition. As a multiple learner system, it tries to exploit the local different behavior of the base learners to enhance the accuracy and the reliability of the overall inductive learning system. There are also hopes that if some learner fails, the overall system can recover the error. Basically the effectiveness of ensemble methods relies on the independence of the error committed by the component base learner. From a general standpoint we know that the effectiveness of ensemble methods depends on the accuracy and the diversity of the base learners. In fact, there is a trade-off between accuracy and independence: more accurate are the base learners, less independent they are. Technically ensembles can enlarge the effective hypotheses coverage, expanding the space of representable functions.